7.0. Reproducibility

What must be recorded to reproduce an agent run?

Deterministic software has one input worth versioning: the code. An agent has at least nine, and any one of them can change the answer without changing the commit. A colleague who says "it worked yesterday" is usually right — a different weight file, a mutated database row, or a re-registered prompt did the work. So an agent release is a tuple, not only a Git commit:

A hosted model name does not freeze provider weights or service behavior. Likewise, qwen3:4b-instruct is a mutable Ollama tag rather than a content pin. Record the installed ID/digest and model metadata from ollama list/ollama show with evaluation evidence. Reproducibility means enough evidence to explain and compare the run, not byte-identical text forever.

The practical test is not "can I replay this token stream" — you cannot, with a sampling model. It is: given this evidence, can I explain the difference between two runs, or prove there is none? Everything below exists to make that question answerable.

How do you select the MLflow destination?

Two environment variables decide where your evidence lands, and both of them are optional.

Start the store first with mise run observability:up, then open http://localhost:5000 to confirm the MLflow server answers. Now run the evaluation from agents/python:

export MLFLOW_TRACKING_URI=http://localhost:5000
export MLFLOW_EXPERIMENT_NAME=agentops-agent
mise run eval:mlflow

Tracking URI: http://localhost:5000

The script falls back to a local file store, in two lines:

_TRACKING_URI = os.environ.get("MLFLOW_TRACKING_URI", f"sqlite:///{Path(__file__).parent / 'mlflow.db'}")
_EXPERIMENT = os.environ.get("MLFLOW_EXPERIMENT_NAME", "agentops-agent")

The prompt version counter in that file is independent of the server's, so a local version 2 and a server version 2 can be different text. This is a real failure mode, not a hypothetical: mise run eval:mlflow loads the root .env, so a commented-out MLFLOW_TRACKING_URI line is enough to split your evidence in two.

How do a trace, a run, a prompt version, and a logged model link together?

The evaluation you just ran wrote its evidence under five MLflow names:

• An experiment is the folder every record lands in — here, agentops-agent.
• A run is one execution stored inside that folder.
• A logged model is the evaluated agent that scorer results attach to.
• A prompt version is one numbered copy of the instruction text.
• A trace is one recorded agent turn, broken into timed steps called spans.

Two independent paths write evidence into the same MLflow server. The runtime path produces traces: the agent exports spans over OTLP, the collector forwards them, and MLflow stores them (7.1. Tracing). The evaluation path produces a run, a logged model, and a prompt version: mise run eval:mlflow writes them directly through the MLflow client.

Nothing joins the two automatically — they meet because both land in the same experiment, by construction:

1. The collector hard-codes the destination experiment id in an export header (x-mlflow-experiment-id: "0" in infra/observability/otel-collector.yaml). It cannot resolve a name.
2. The evaluation script resolves an experiment by name: _EXPERIMENT = os.environ.get("MLFLOW_EXPERIMENT_NAME", "agentops-agent").
3. The course MLflow image closes that gap before the server starts. infra/mlflow/entrypoint.py idempotently renames built-in experiment 0 to MLFLOW_EXPERIMENT_NAME — its docstring states the intent exactly: "Give experiment zero the course-wide name without splitting lineage."

Point either path somewhere else and the join silently disappears: traces in experiment 0, evaluations in a new experiment 1, and no error anywhere. The whole lineage graph a release note needs looks like this:

flowchart TD
    Commit[Git commit] --> Instruction[INSTRUCTION in composition.py]
    Instruction --> Pin{AGENT_PROMPT_URI set?}
    Pin -->|"yes"| Load[load exact pinned version]
    Pin -->|"no"| Match{identical registered<br/>template exists?}
    Match -->|"yes"| Reuse[reuse newest matching version]
    Match -->|"no"| Register[register one new version]
    Load --> Version[prompt.uri + prompt.version]
    Reuse --> Version
    Register --> Version
    Version --> Model[initialize_logged_model<br/>params: agent_model, prompt_uri, prompt_version]
    Model --> Run[start_run: eval-prompt-vN<br/>tags: prompt_name, prompt_version]
    Run --> Evaluate[genai.evaluate<br/>model_id = logged_model.model_id]
    Evaluate --> Metrics[Metrics + eval traces]
    Metrics --> Finalize[finalize_logged_model<br/>READY or FAILED]
    Runtime[Runtime agent turn] -->|OTLP| Collector[OTel Collector<br/>x-mlflow-experiment-id: 0]
    Collector --> Experiment[(Experiment agentops-agent)]
    Evaluate --> Experiment

The arrow that does not exist is worth naming: a production trace carries no prompt version. The runtime uses the committed INSTRUCTION, so the Git commit of the deployed image is what attributes a production trace to a prompt — not the registry.

How does MLflow link prompt and model evidence?

One command produces the whole chain: mise run eval:mlflow.

It first selects the exact prompt that each fresh evaluation agent will load. A pinned run loads an immutable numeric AGENT_PROMPT_URI; an unpinned run reuses the newest registered version with identical committed text, searching older pages when needed, and registers only when no template matches. Mutable aliases are rejected because they could change during one long run. It then initializes a logged model, evaluates the full conversations, and marks the model READY or FAILED.

The evaluator sets _TRACKING_URI before it imports agent.composition. That module constructs the ADK discovery root immediately, and its fresh-agent factory later loads pinned prompts from the same store. The ordering also keeps eval:mlflow, eval:cost, eval:ground, and eval:ab children on evals/mlflow.db when no tracking URI is configured.

The selection and lineage fields in mlflow_eval.py are:

def _evaluation_prompt() -> PromptVersion:
    if settings.prompt_uri:
        return mlflow.genai.load_prompt(_prompt_selection())
    if matching := _matching_registered_prompt(INSTRUCTION):
        return matching
    return mlflow.genai.register_prompt(
        name=_PROMPT_NAME,
        template=INSTRUCTION,
        commit_message="AgentOps Agent system instruction",
    )

prompt = _evaluation_prompt()
model_params = {
    "agent_model": settings.model,
    "agent_model_provider": settings.model_provider.value,
    "prompt_uri": prompt.uri,
    "prompt_version": str(prompt.version),
}
if model_digest := os.environ.get("EVAL_MODEL_DIGEST"):
    model_params["agent_model_digest"] = model_digest

logged_model = mlflow.initialize_logged_model(
    name="agentops-agent",
    experiment_id=experiment.experiment_id,
    model_type="agent",
    params=model_params,
)

experiment_id comes from mlflow.set_experiment(_EXPERIMENT) on the line above, which pins the logged model into the same experiment the traces land in. model_type="agent" tells the MLflow UI how to render it. The scheduled workflow resolves the Ollama digest into EVAL_MODEL_DIGEST; a manual topology should supply that value when it can resolve one.

The evaluation itself closes the loop by passing model_id, so every scorer result attaches to that logged model rather than floating in a run:

cases = _load_cases()
observed = {}
with mlflow.start_run(run_name=f"eval-prompt-v{prompt.version}"):
    mlflow.set_tags({"prompt_name": prompt.name, "prompt_version": str(prompt.version)})
    predictor = mlflow.trace(_recording_predictor(observed), name="agentops_eval_case")
    with _without_mlflow_prediction_probe():
        result = mlflow.genai.evaluate(
            data=cases,
            predict_fn=predictor,
            scorers=_scorers(),
            model_id=logged_model.model_id,
        )
    _write_model_observations(
        observed,
        cases,
        resolved_prompt_uri=prompt.uri,
    )

The recorder preserves the exact outputs that MLflow scored, so the scheduled cost and grounding steps can evaluate the same answers without thirty duplicate conversations. The predictor creates its trace explicitly and disables MLflow's usual one-sample trace probe, because that probe would be a real extra model generation.

In the scheduled workflow, the capture is bound to the provider/model digest, prompt selection, normalized eval contract, and source revision. The workflow also fixes the Ollama serving window at 8,192 tokens and sampling temperature at zero, then records the context, temperature, model digest, and Ollama version in model.json. ADK's supported GenerateContentConfig forwards that temperature to the OpenAI-compatible request. This greedy setting lowers sampling variance; it cannot make model execution bit-reproducible.

The cost observation and baseline retain both files' provenance. Comparison fails closed when the prompt, eval contract, provider/model/digest, context window, Ollama version, or temperature changes. The recorded source revision identifies where the baseline came from but may differ on the candidate revision whose cost drift you are measuring. A local eval:mlflow run writes a capture too, but reuse fails closed unless EVAL_MODEL_DIGEST and GITHUB_SHA identify its model artifact and source revision.

The try/except around that block finalizes the logged model FAILED on any exception — including a deterministic metric falling below its threshold in _required_metric_failures — and READY only on a clean pass. That matters for reproducibility more than it looks: a logged model left in READY is a claim that its evidence was produced, not merely attempted. Never promote a model whose lineage ends in FAILED.

How do I know which prompt produced this behavior?

By default every runtime uses the committed INSTRUCTION from composition.py — the prompt is whatever Git says it is at that commit. In the host development/evaluation environment, you can compare behavior against a registered version by setting AGENT_PROMPT_URI:

export MLFLOW_TRACKING_URI=http://localhost:5000
export AGENT_PROMPT_URI=prompts:/agentops-agent-instruction/2
mise run eval:mlflow

A prompt URI is the registry address of one exact version: prompts:/<name>/<version>.

_instruction() then loads that exact version from the self-hosted MLflow prompt registry at startup, so traces and evaluations from that host process are attributable to one immutable prompt text. Configuration validation rejects a malformed URI at startup, and mlflow is imported lazily from the dev dependency group.

The production agent image intentionally installs no dev dependencies, so it does not support AGENT_PROMPT_URI; containers and Kubernetes use the committed instruction. This keeps the runtime image small and avoids an MLflow availability dependency on startup. Promote or roll back production behavior by shipping the evaluated Git/image version, not by setting a registry URI in the deployment.

Version a prompt through the evaluation script rather than a separate registration step. MLflow's register_prompt always creates a version; the repository adds exact-template reuse around it. _matching_registered_prompt() reuses the newest identical template, paginates through older versions when the latest differs, and calls register_prompt only when no version matches.

The full workflow is four steps:

1. Edit INSTRUCTION in composition.py (for example, tighten an operating rule).
2. Run mise run eval:mlflow; it reuses an identical registered template or creates the next version for changed text, then evaluates it in a run named after the selected version.
3. Every run carries prompt_name and prompt_version tags, so in the MLflow UI you filter or sort the experiment by tags.prompt_version and compare v1 and v2 scorer results side by side.
4. Decide with evidence: keep the new version if the deterministic scorers hold and any judge evidence improves, otherwise roll back.

Exact-template reuse cuts both ways. Change whitespace and you get a new version that may behave identically. Change nothing and re-run, and you get a second run named eval-prompt-v1 against the same version. Revert to text from an older version and the helper reuses that historical version instead of creating a duplicate.

That second case is useful and confusing at once. It is the cheapest way to measure your own sampling noise, and it is also the most common cause of "why is there no v2?" Check the run name the script prints against the edit you thought you made.

The registry is the self-hosted MLflow server only; there is no hosted prompt service in this course.

How do you test a prior prompt version?

Point a host development/evaluation process at the prior registered version:

export MLFLOW_TRACKING_URI=http://localhost:5000
export AGENT_PROMPT_URI=prompts:/agentops-agent-instruction/1
mise run eval:mlflow

Use that comparison to decide which committed version to build. Production rollback means redeploying a previously recorded, known-good image digest and its matching source commit. Prompt evaluation supports commit selection; image scan and smoke evidence must cover the artifact built later. Alternatively, revert INSTRUCTION, re-evaluate, and release a corrected image. Leave AGENT_PROMPT_URI unset: a pinned host process depends on MLflow at startup, while the production path deliberately avoids that coupling.

Evolve the eval set with the prompt, not after it. A prompt change that adds behavior (say, a new escalation rule) needs cases that exercise it, or v1 and v2 will score identically while behaving differently.

Record the dataset commit next to the prompt version, as the closing checklist on this page asks. The dataset is agents/python/evals/ops.evalset.json, and comparing eval-prompt-v1 with eval-prompt-v2 is only valid on the same eval set commit. If the dataset changed in between, re-run the old prompt version against the new dataset before drawing a conclusion.

How is data reset for comparison?

Reset the disposable runtime state before any comparison you intend to trust.

cd agents/python
mise run data:reset

That deletes runtime sessions/tasks and the writable incident copy, then the next run initializes from the committed seed. Record the seed Git commit and do not compare a fresh run with one whose mock actions changed service/incident state.

The reason is concrete. restart_service, resolve_incident, and save_incident_note write. Once a case resolves INC-001, a later run asking "what is the status of INC-001?" is answering a different question against a different database. It will score differently for a reason that has nothing to do with your prompt.

The separation is by design. _DEFAULT_DATA_DIR in config.py points at the committed, immutable dataset, while _DEFAULT_STATE_DIR points at a disposable .state directory. That is why mise run data:reset is a one-line rm -rf .state: nothing under Git ever needs restoring.

mise run eval:mlflow is stricter than a manual run and does not depend on you remembering the reset — see the next section.

What is deterministic and what is not?

Two lists decide what you can pin and what you can only record.

Deterministic: types, validators, SQL seed, retrieval ranking, tool functions, policy callbacks, graph topology, scanner/test configuration, and exact expected trajectories.

Non-deterministic/external: model output, provider implementation, sampling, network timing, Spot scheduling, and live service availability. Pin/control what you can, record what you cannot, and use distributions/evaluations instead of promising replay identity.

def ask(turns: list[str], eval_id: str) -> dict[str, Any]:
    """Run one conversation with an isolated user and disposable runtime state."""
    require_attributable_runtime()
    with _EVAL_STATE_LOCK, isolated_state(f"agentops-{_eval_user_id(eval_id)}-"):
        evaluation_agent = build_conversational_agent()
        return asyncio.run(_run_disposable(turns, eval_id, evaluation_agent))

What remains uncontrolled is the model itself: the same prompt, the same state, and the same seed can still produce a different trajectory. Run the eval twice before you attribute a metric change to your edit.

Which comparisons are silently invalid?

Every failure mode below produces a number. None of them produce an error. That is what makes them expensive: you ship a prompt because v2 scored better, when what actually changed was the dataset.

flowchart TD
    Start[Two eval runs to compare] --> Evalset{Same ops.evalset.json<br/>commit?}
    Evalset -->|no| Invalid[Label the comparison invalid<br/>re-run the baseline]
    Evalset -->|yes| Store{Same MLFLOW_TRACKING_URI<br/>and experiment?}
    Store -->|no| Invalid
    Store -->|yes| ModelPath{Same provider, model, base URL,<br/>and resolved digest?}
    ModelPath -->|no| Invalid
    ModelPath -->|yes| Judge{Same three MLFLOW_JUDGE_*<br/>settings on both runs?}
    Judge -->|no| Partial[Deterministic metrics comparable<br/>judge metric is not]
    Judge -->|yes| Valid[Comparable: attribute the delta<br/>to the prompt version]

How do you capture the release tuple in one pass?

The table at the top of this page is only useful if capturing it is cheap. Each row has a command that prints the evidence:

git rev-parse HEAD                        # Code
git status --porcelain                    # Code: must be empty, or the commit is a lie
ollama show qwen3:4b-instruct             # Model path: local weight digest and parameters
cd agents/python && mise run config:check # Model path, prompt, data: resolved settings, secrets masked

mise run config:check is the highest-value line. It constructs Settings exactly the way every runtime entrypoint does and prints every resolved field. So model, openai_base_url, prompt_uri, data_dir, and state_dir are captured as the process actually sees them — not as your .env suggests they might be.

Dependencies need no command: uv.lock and mise.lock are in the commit you just recorded.

# Container: the digest actually running, not the tag Skaffold wrote
kubectl -n agentops get pods -l app.kubernetes.io/name=agentops-agent \
  -o jsonpath='{.items[*].status.containerStatuses[*].imageID}'

# Runtime: the fully rendered overlay that produced the cluster state
kubectl kustomize infra/k8s/overlays/local

What proves this page worked?

Prove the lineage end to end, once:

1. Start the observability stack (mise run observability:up) and confirm http://localhost:5000 serves the agentops-agent experiment.
2. Run mise run data:reset, then mise run eval:mlflow with MLFLOW_TRACKING_URI set. Check the printed tracking URI matches the server, and that no Local UI: suggestion appears.
3. In the MLflow UI, open the eval-prompt-vN run, read its prompt_name/prompt_version tags, follow them to the logged model, and confirm its agent_model, prompt_uri, and prompt_version params and its READY status.
4. Launch the agent with tracing configured — set the OTLP environment from 7.1. Tracing (chiefly OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318; without it setup_telemetry() exports nothing), then from agents/python run mise run run and send one ordinary turn. Find its trace in the same experiment. Two paths, one lineage — that is the join this chapter exists to protect.

Those steps produce most of the evidence. Copy this checklist and fill it in before you compare two runs:

1. Code commit: git rev-parse HEAD, with git status --porcelain empty.
2. AGENT_MODEL_PROVIDER: printed by mise run config:check.
3. AGENT_MODEL: printed by mise run config:check.
4. OPENAI_BASE_URL: printed by mise run config:check.
5. The Ollama model ID when local: from ollama show qwen3:4b-instruct.
6. The gateway profile, when the model call went through a gateway.
7. Prompt URI and version: prompt_uri from mise run config:check, or the run's prompt_version tag.
8. Dataset commit: the Git commit of agents/python/evals/ops.evalset.json.
9. Image tag and digest: the imageID from the kubectl command in the collapsible above.
10. MLflow run id and logged-model id: from the eval-prompt-vN run in the UI.

Reset state and use the same eval set; otherwise label the comparison invalid.

You are done when:

• http://localhost:5000 serves an experiment named agentops-agent.
• mise run eval:mlflow ended with Tracking URI: http://localhost:5000 and printed no Local UI: suggestion.
• The eval-prompt-vN run carries prompt_name and prompt_version tags, and its logged model is READY.
• One ordinary agent turn shows up as a trace in that same experiment, beside the evaluation run.
• Every line of the checklist above has a value written next to it.

Continue to 7.1. Tracing when one evaluation run and one runtime trace sit in the same agentops-agent experiment.